by K.Nikhil
Project Info
Street Names:-
Zip Codes:-
Here we provide some overview of the dataset
statistics of the dataset
The code will be at the end of notebook
I looked at users contributed to hyderbad OpenStreetMap most frequently. The top three users are:
I also looked at the top cuisines: indian,regional,vegetarian,chinese,coffee_shop,international,ice_cream,pizza,indian,_chinese,_hyderabadi,veg_and_no resaturant,cafe,tiffins,South_Indian,asian;noodles;chinese,sandwich,chicken,south_Indian,Chinese,Fast_Food,american,sushi
I also looked at the Universities :
AMR-APARD.Engineering College,Law College,Department of Physics,National Institute of Rural Development,Department of Chemistry,Computer Science Department,University Library,Department of Bio Chemistry,Tagore Auditorium,Jawaharlal Nehru Architecture and Fine Arts University
import xml.etree.cElementTree as ET
import pprint
import re
import codecs
import json
import collections
import pymongo
import os
datadir = "data"
cal_data = "hyderabad_india.osm"
#Parse through the file with ElementTree and count the number of unique element types to understand overall structure.
def count_tags(filename):
tags = {}
for event, elem in ET.iterparse(filename):
if elem.tag in tags:
tags[elem.tag] += 1
else:
tags[elem.tag] = 1
return tags
cal_tags = count_tags(cal_data)
pprint.pprint(cal_tags)
Using Key_type and process_map we can check weather the keys are valid in mangoDB,and we will check if there are any potential problems.
Using the function 'key_type', I counted three tags:
import string
from pymongo import MongoClient
lower = re.compile(r'^([a-z]|_)*$')
lower_colon = re.compile(r'^([a-z]|_)*:([a-z]|_)*$')
problemchars = re.compile(r'[=\+/&<>;\'"\?%#$@\,\. \t\r\n]')
def key_type(element, keys):
if element.tag == "tag":
for tag in element.iter('tag'):
k = tag.get('k')
if lower.search(k):
keys['lower'] += 1
elif lower_colon.search(k):
keys['lower_colon'] += 1
elif problemchars.search(k):
keys['problemchars'] += 1
else:
keys['other'] += 1
return keys
def process_map(filename):
keys = {"lower": 0, "lower_colon": 0, "problemchars": 0, "other": 0}
for _, element in ET.iterparse(filename):
keys = key_type(element, keys)
return keys
cal_keys = process_map(cal_data)
pprint.pprint(cal_keys)
The problem that we encountered in this dataset are street name abbreviation and few zipcodes. In this following code, we build the regex which will be used for matching the last element in the string. Then we come up with a list of mapping that need not to be cleaned.
audit_street_type function
will search the input string for the regex and searches if key is present or not if not we will add itaudit function :-
will return the list that match previous two functions. After that, we would do a pretty print the output of the audit. zip_update :-
Checks for no in string if it starts with 5 it will be updatedWith the output list .we can understand and fill-up our "mapping" dictionary as a preparation to convert these street name into proper form.
from collections import defaultdict
street_type_re = re.compile(r'\b\S+\.?$', re.IGNORECASE)
expected = ["Avenue", "Boulevard", "Commons", "Court", "Drive", "Lane", "Parkway",
"Place", "Road", "Square", "Street", "Trail"]
mapping = { "Ct": "Court",
"St": "Street",
"st": "Street",
"St.": "Street",
"St,": "Street",
"ST": "Street",
"street": "Street",
"Street.": "Street",
"Ave": "Avenue",
"Ave.": "Avenue",
"ave": "Avenue",
"Rd.": "Road",
"rd.": "Road",
"Rd": "Road",
"Hwy": "Highway",
"HIghway": "Highway",
"Pkwy": "Parkway",
"Pl": "Place",
"place": "Place"}
def audit_street_type(street_types, street_name):
m = street_type_re.search(street_name)
if m:
street_type = m.group()
if street_type not in expected:
street_types[street_type].add(street_name)
def is_street_name(elem):
return (elem.attrib['k'] == "addr:street")
def audit(osmfile):
osm_file = open(osmfile, "r")
street_types = defaultdict(set)
for event, elem in ET.iterparse(osm_file, events=("start",)):
if elem.tag == "node" or elem.tag == "way":
for tag in elem.iter("tag"):
if is_street_name(tag):
audit_street_type(street_types, tag.attrib['v'])
return street_types
cal_street_types = audit(cal_data)
def update_name(name, mapping):
for key in mapping:
if key in name:
name = string.replace(name,key,mapping[key])
return name
def update_zip(zipcode):
checker = re.findall('\d+', zipcode)
if checker[0].isdigit():
if re.match(r'^\d{6}$',checker[0]):
zipc=zipcode
return zipc
def audit_zipcodes(osmfile):
# iter through all zip codes, collect all the zip codes that does not start with 02
osm_file = open(osmfile, "r")
zip_codes = {}
for event, elem in ET.iterparse(osm_file, events=("start",)):
if elem.tag == "node" or elem.tag == "way":
for tag in elem.iter("tag"):
if tag.attrib["k"] == "addr:postcode" and not tag.attrib["v"].startswith("50"):
if tag.attrib["v"] not in zip_codes:
zip_codes[tag.attrib["v"]] = 1
else:
zip_codes[tag.attrib["v"]] += 1
return zip_codes
zipcodes = audit_zipcodes(cal_data)
"""for zipcode in zipcodes:
print zipcode, zipcodes[zipcode]
for cal_street_type in cal_street_types:
print cal_street_type,cal_street_types[cal_street_type]"""
Few errors are there in zip codes and street names all errors will be updated by update_zip,update_name functions
The function to process the XML file, make ready for insert into MongoDB
CREATED = [ "version", "changeset", "timestamp", "user", "uid"]
def shape_element(element):
node = {}
node["created"]={}
node["address"]={}
node["pos"]=[]
refs=[]
# we only process the node and way tags
if element.tag == "node" or element.tag == "way" :
if "id" in element.attrib:
node["id"]=element.attrib["id"]
node["type"]=element.tag
if "visible" in element.attrib.keys():
node["visible"]=element.attrib["visible"]
# the key-value pairs with attributes in the CREATED list are added under key "created"
for elem in CREATED:
if elem in element.attrib:
node["created"][elem]=element.attrib[elem]
# attributes for latitude and longitude are added to a "pos" array
# include latitude value
if "lat" in element.attrib:
node["pos"].append(float(element.attrib["lat"]))
# include longitude value
if "lon" in element.attrib:
node["pos"].append(float(element.attrib["lon"]))
for tag in element.iter("tag"):
if not(problemchars.search(tag.attrib["k"])):
if tag.attrib["k"] == "addr:housenumber":
node["address"]["housenumber"]=tag.attrib["v"]
if tag.attrib["k"] == "addr:postcode":
node["address"]["postcode"]=tag.attrib["v"]
node["address"]["postcode"] = update_zip(node["address"]["postcode"])
# handling the street attribute, update incorrect names using the strategy deve
if tag.attrib["k"] == "addr:street":
node["address"]["street"]=tag.attrib["v"]
node["address"]["street"] = update_name(node["address"]["street"], mapping)
if tag.attrib["k"].find("addr")==-1:
node[tag.attrib["k"]]=tag.attrib["v"]
for nd in element.iter("nd"):
refs.append(nd.attrib["ref"])
if node["address"] =={}:
node.pop("address", None)
if refs != []:
node["node_refs"]=refs
return node
else:
return None
# process the xml openstreetmap file, write a json out file and return a list of dictionaries
def process_map(file_in, pretty = False):
file_out = "{0}.json".format(file_in)
data = []
with codecs.open(file_out, "w") as fo:
for _, element in ET.iterparse(file_in):
el = shape_element(element)
if el:
data.append(el)
if pretty:
fo.write(json.dumps(el, indent=2)+"\n")
else:
fo.write(json.dumps(el) + "\n")
return data
data = process_map(cal_data, True)
client = MongoClient()
db = client.BostonOSM
collection = db.bostonMAP
collection.insert(data)
collection
os.path.getsize(cal_data)/1024/1024
os.path.getsize("hyderabad_india.osm.json")/1024/1024
collection.find().count()
len(collection.group(["created.uid"], {}, {"count":0}, "function(o, p){p.count++}"))
collection.find({"type":"node"}).count()
collection.find({"type":"way"}).count()
pipeline = [{"$group":{"_id": "$created.user",
"count": {"$sum": 1}}},{"$sort": {"count": -1}},{"$limit": 5}]
for res in collection.aggregate(pipeline):
print "Count : "+str(res["count"])+" "+"id : "+str(res["_id"])
pipeline = [{"$group":{"_id": "$created.user",
"count": {"$sum": 1}}},
{"$project": {"proportion": {"$divide" :["$count",collection.find().count()]}}},
{"$sort": {"proportion": -1}},
{"$limit": 3}]
for res in collection.aggregate(pipeline):
print "id : "+str(res["_id"])+" "+"proportion : "+str(res["proportion"])
pipeline = [{"$match":{"amenity":{"$exists":1}, "amenity":"restaurant", "cuisine":{"$exists":1}}},
{"$group":{"_id":"$cuisine", "count":{"$sum":1}}},
{"$sort":{"count":-1}},{"$limit":20}]
for res in collection.aggregate(pipeline):
print "Count : "+str(res["count"])+" "+"id : "+str(res["_id"])
pipeline = [{"$match":{"amenity":{"$exists":1}, "amenity": "university", "name":{"$exists":1}}},
{"$group":{"_id":"$name", "count":{"$sum":1}}},
{"$sort":{"count":-1}}]
for res in collection.aggregate(pipeline):
print str(res["_id"])
Ideas to improve data quality of OSM data :
when we did uditing to data the thing which we saw was data is having only minor errors caused by human input there are 886 users so there is chance to get human errors in this project.
I reccomend to cross check with the street names and pincodes by asking remaining users if one did wrong reminaing 885 users will surely correct the wrong part.
There must be leader board like kaggle and stackoverflow since it is leaderbaord based we will get quality outputs from contributers.
There are so many another areas wich the map didnt cover like universities (IIT hyderbad),So there must a person who will keep tasks like to complete one part of hyderbad this week and he will have leaderboard it must work same like kaggle and stackoverflow .
we can resolve this issue by cross-referencing/cross-validating missing data from other database like Google API.
Potential cost of the implementation:
There're few potential issues could you see that may arise from the implementation of this solution. One of which is the amount of effort to engineer all these processes and the cost of creating, auditing & maintaining these initiatives could be so overwhelm and require a dedicated team responsible for all these projects.